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ABSTRACT ' *" • ■ ' 

The H&tional Assessment of Educational Pr'o.gress; 
Second Science Assessment Ho-Shbw Study ass,essed the magnitude and 
causation- -of sonresponse biases. A Ho-shov is defined a« an 
individual wHo wis selected. as a sample 'respondent b^t failed to be 
pces^nt for regular assessment' of the 17-yeai-old group. The 
procedure whereby a sample of -eligible 17-year-oia Ho-Shows were 
•selected to take four specific Ho-Show assessment packages is briefly 
.de&criJied. -Estimates of biases due to nonhresponse 'were made for the . 
following domain variables; region; sex; race; size and type o£ 
community: derive^ 'parental educatiot^. 7he%e domains are outlined. 
Also docjumented are the <4omain estimation meth.odolo.gle*s, terminology 
and the jiethods- utilized tot the computation of formtlas. Ho-Show 
domain analysis results axe brielCly summarised in tables^ but are 
detailed aore fully elsewhere is separate appendixes,. Results dhow 
that reliability estimates were positive and^sevecal 'sere 
significant, ^n^^-Cating that regular assessment students generally 
performed 'better on assessil^ent packages than did the lio-shows. 
Separate exercise analyses of bias are cpni^idered in section 6 -as t^n , 
extension of the- analyses covered by the study. Primary type of 
information provided by rewrt; Procedures (Evaluation) fSaiplingl. 
(Author/AEPi ■ \ . , 
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7 1. BACKGROUND AND OBJECTIVES • 

TfiB aualTsls of National* Assessment No*Shov3 Cite^, nonrespondents to 

package admlnlstratlonsl was undertaken to assess the loa^itude of associated . 

' * 4 

non/esponse biases as well as to determine the origins and causation of these 

biases. Field operations were conducted both La-School and Out -of -School* A 

• * * * 

total of 1990 ^ackag^es vere administered during the In-School portion of the 

study. AuxUla^ data on students^ attendance fecord^s,^ jpourse grade?) and 

curricula v^re collected on 1753 Age Class 3 regular Assessment paticipants 

dad 1324 No-Shows. Packages* were also ^administered Out-of-School to 130 of 

the 598 No-Shows who, could not be locked during an In-School followup. ^ually 

supplemei^tjary data I V regarding the reasons for sti^ent abs^eilleeism from 

regular assessment administrations, werft ascertained from 1989 ,In-School 

No-ihowa and 13 O^Oat-of -School No-Shows. 

For the purposes of this study a No-Shpw is^deflned as an individual 

who was initially selected' as a jtaniple respoxKlent t^ut failed, to be present 

for, seventeen-year-old regular assessment.. The No-Show study .was con^cted ^ 

for sevenfeen-ytar-old respondents only since it wis felt that the npnreaponse 

• - ' ■ ^ ' ' * - / 

problem was most serious at that age level." A multistage, probability sample 

wa^s selected from the H&EF*7ear 04 In-School sample. Briefly, the sample was 

selected as follows^ ?lrst, a sai^Ze^of primary units was selected from the 

KAEP Year OA In-ScKool prlinary sampler Secondly, within o^ch selejfxted No-Show 

FSU» those schools eligible to take a Ko-Show package were determined. Thirdly » 

^ ' I. 
within these eligible schools, eligible 17«^year-old No-Shows^ were selected to 

« • *' 

take specific' No-Show packages. Finally, a systematic sample of. .these In-Scliool 



No-*Sfiov3 wlio cotild not contacted^ txi Scliool was selected for an Out-of- * 
School foilowup study. Further details of saapllng and field procedures' 
are presented elsev^iere Il]« Procedures for coctputing sample v^^ights from 
the No-*Shotf study saisple design have also Been presented [2] , 

Pour packages ware seleqted /or the No-Shov studyl The four packages 
Included three group packages, selectcid'from the eleven Year 04 group 
packages; and one,*£ndlvldtial pacJcage,; selected from the three Year 04 - 
indlvldtial packages* The selected group packages were munbers 1, 3, and 9 
and the selected Individual package was numBer 13 « 



Auxiliary and supplementary data restilts have been documented [3] . 
Among several other sosslBld oBservatlons are the following: 



Cl) 'No**Shows, as compared to regular assessment students, . 
are absent from school more often, get lower course 
grades, are led? inclined to enroll in college pre-^ 
paratory curricula, and toroll In fewer courses; 

(2) About H]f of In-School No-Shows claltaed that they 
were not notified^ about the regular assessments 
administration;* 

(3) About l^^^ f Cot those^In-Scbool Ho-Shows^ whd^ remembprcd) 
. we're absent from school *on their original reoflar 

assessment administration data; " 

C4) The looming jjuDurs of the day were clearly preferred 
'.for regular assessment sessions; 

,C5) About half of the Out-of-School No-Shows wejie not 
enrolled in the school where HASP had assumed tilAt 
they were enrolled s ^ ' \ 

(6) A moclerate proportion of t.hqffc, In-School No-Shows \ . 
^ vbo skid they were not aBsent from school on ishe \ 
' day of regular assessment Indicated that* they \ 

/ ' other school-cri^ted commitments ^ \ 



" X discussion of jietliodology axd a presentatloA o£ accompanylhg estimates 
, jof package composite Biases aW rel*-Biases have Been documented {4-5] . Host 
estimates were positive and several were significant, Indicating that^^^lar 
assessment students generally performed Better on these packages than did the 

\ 

No-Shbws. The ahove-mentloned auxiliary data .results tend to parallel these 
findings* Wltl the group packages, the magnitude of bias was smaller and 
statistically significance less often ^rtien only In-School No-Shows were 
involved in- the <y>mputatlons. 

The overall purpose of the j^c^ent document willf be to present further- 
methodology and results, Speclilcally» estimates of biases due to nonresponse 
are made for several domains or subpopulatlons. The domains considered here 
• are formulated from several variables which are of Intetest to NAEP and which 
have been determined for each- s^pled regxilar assessment and JSo-Show student. 
■ The variables considered here are as follows: • ^ .'^ 

.'■(i) Region; * , * 

. ' (pt) Sax; * * 

(3) -Race;, • • 

C4) Size and Type 'of Comaunlty C3XQC) ; ' • * 

C3}«^erlved Parental^ Education. ^ < 

* A discussion of the specific domains, formed as marginal categories of the 
above variables, is presented in section 2 of this repprt» 

of mathematics ot science exercises in the package that are answered correctly 
^ by a regular assessment or Ko-Stiow respondent. -^The total number, of exercises 



0 



associated' with, tlie oathematlcs and science portions of the three group 
packages (01% 03, &nd 0*91, that are considered in these analyses, is 
presented In tatle 3 of "appenddbt B." 'This measurement variable yields a 
stople average of the separate-ex^clSe B-values involved. Similar separate- 
exercise analyses .of Bias will Be considered as an, extension ,of the present 
analyses (see section 6.2}. ' . 

« 

It shoxild also be mentioned Initially that j)nly thp^No-Show study 
group packages are' considered in these analyses. la other Aords, No-Show, 
Study individual package. 13 will not he include4 in these analyses. It Is 
^excluded because of th^ craaplex nature of the administration of the pacKage. 
More specif Icaliy, tt contains a liixge 'amount of conditional braijchinj among, 
pertain exercises making the identification of "Icey" exercises for evaluation 
a 41fflcult ifask. . On J:he other hand, exercises of the group packages are 
straight-forward and "key" exercises can be easily identified for evaluation 



purposes. 



•i 



2, BESCRlfTXON OP DOMAINS 



Results for tfie 'National Assessmeat df Educational Progress surveys • 

■ . . : ' ' ' 

are gubltsh^ separately for eactAge Class— 9-7ear-old8 , 13-year-olds, 

17-ycar-oid8, and young aduits aged i6 to 35, JfitbSxi each o$ t^se Age 

Classes, results are Reported for each, of the five .groups or domafns. The 

donaltf^varlables are sex, region of country, race, Sfee^and Tjrpe of Coraminity ^ 

CSTOC) , und derived "parental educafJLon, The domain variables along .with the 

reporting categories for each dodaln variable are listed in table 2U. The 

tables in appendix A compare 17-year-old regwla* assessaept respondents with 

17^ear-old noiaespondents or No-Shtfcrs by domain. The nonrfespondents fcould 

have been contacted either through^th^ In-School or Out-of-School portion of 

the No-Show study. The No-Show sample and data collection activities have 

♦ - • * * . ' -k' 

been documented elsewhere lU* , "tl * 'N\^ 

■2,1 Sex Domains . , 



The twa sexes, 4ale and female, are the reporting categorJ.es for the s 

domalm Prom appendix table,A.l, it can be seen, in general, that there are 

' i ♦ • . 

an approximately equal number of male and female respondents and nonrespondents* 

2.2 Region DoiBalng . . ' . 

The four regional r^ortlng categories are those regional groupings 
defined by the Office of Business Economics, Department of Commerce. The' 

four regional grouplilgs by State aye defined In table 2.2^ Prom appendix 

' ' ' . " 

table A. 3, it can be seen tha^ the regular respondents arp fairly equally - 

V' ' • 

divided between the four regions. Noorespond^nts, on tbk other hand, are 
. slightly more concentrated In the. Central and Western regions. This event * 



Table 2^1. Domain variables and reporting categories for 
National Assessment o£ Educational Progress 



Domain variable 



Reporting categories' 



Sex 



Region 



Rac£ 



Size and Type of Community (STOC) 



: Derived parental education 



Male 
Female 

Northeast 
^ Southeast 
Central 
West 

White 
Black . 
Other^ 

Extreme' ruc^al 

Low. metropblltan 

High metropolitan 

Inner city fringe ' 
^ Suburban fringe 
* Medium city 

Small city 

No high school 
Some high school ' ' 
Graduated frodi hlgh.schqol 
Post high school 



12 0 • .'■ 



'4 



• 



4? 



' r 

i 
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OHWITTED DOE TO CONFIDENTIALITY .x 
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r 



»• 



occurred because of t&a refus^ of spveral schools In tha Horthi^ASt and 

Soutae^St to release t&e namea and .aadresaes of nonrespondents who had 
' - \ ' \ • ' . . 

not beexr contacted during t&e lii'^SctooX , portion of t!ie No-Stiow stidy, 

^ . *^ ■ ^ 

2^3 ^ce domains • / '"^ , * 

Race reporting categories are white, blacl^r, and other* The "other 

category IncXx^es Puerto Ricans, Hexlcans, Orientals , ^olyneslai^, Asians, 

American Indians, etc* Observing appendix table A»2', it can be. seen that 
* « <T ft 

t approximately 70 pjercent of the sample respondents and nonrespondents are 

white, approximat^y 15 percent are black, and approsdmately 10 percent 

are. classified as other* * ' * ^ . ' .1 ' 

' < ' ' \ • ^ * 

2»4 ^ STOC Domains- * ^ . " ^^^Mj^ 

The seven STOC categories are a means of classifying respondents by 

characteristics of their home comsmnity such, as si;e, location,^ and o^a^^u pa- 

^ * - " * • "* *■ 

tion of residents. The first two categories, extreme rural and low metro*- 

polltan, ^jCj^v^raampled to obtain enough, re^ondents to make domain ^ 

estimate^^!|^nP||fhe desired .precision, A sufficient number of tespgnden'ts^ 

In the *third category, itlgh. metropplitan, are obtained by normal sdm{>llng ^ 

procedures to make domain estimates with, the desired precision. A detailed 

< ]^ ^ . ^ ■ ^ - ^ 

explanation of ^he definition, formation, and ^izie of STOC categories is 

^ • • 
provided qlsewhere 16] » Observing appendix table A,4, one sees that appi^ojc* 

Imately AO percent of the sample respondents nonresjpondents are classified 

in the siSialT city STOC category; approximately 10 percent are classified in 

^ch. of the remaining six STOC categories* ^ . 

^ * * • f , ' ^ 

. \' *' . ■ A / ' * 



2.5 Derived Parental Educatltm Homalns ^ 

Delved parental education codes vc^e determined By comparlxlg the highest ^ 

grade of school *cotiipleted for both parents and selecting the highest grade 

among both parents. If this highest grade completed was at mo^t 8» then the 

derived parental education classification was no hlgfi school; if this highest 

grade completed was at least 9 but less than 12 » then the class;Lf icatlon vas 

some high school; if' this highest grade completed was 12, then tb^ classlf icatdCbn 

• • • « * 

vas graduated froa high school; and finally ^ If this highest grade completed 

was greater than 12 » then the classification was post high school. Referring. 

to appendix table A.S^ it can be noted that approximately 10 percent of the 

sample »spondents and nonrespondents did not answer this quesiton ei£her because 

thay.di^not know:. ^hev highest education level of. both parents they refused d 

or forMt^to answer the.question^ or" because bf a specific State law prohibiting 

re&rearch organisation ^rom asking a student..thls question* Approximator^ 

10 percent of the sample respondents and nonrespondents hkd derived parental 

.education classifications of no high school; approxlmatj47 1^ percent were 

classified ^s some high school; about 30. percent were classified as gradtiated 

from high school; and about 40 percent were classified as i^st high schools 



. 3, .DESCRIPTION • 0? WETHODOLOCt 

3.1 DeflJiltloiis and Terminology 

Tha' notation tfsed In this paper Is similar to tbat used, In a previous 
paper which tepcfted Bias estWtes Bj- suBject and package" as found in tha 
No-?Shoy stu^j*-. ThlS'paper was presented at the August 1974 meeting of the ^ 

Merican Statistical Assoclatilon 15] . ' * ' ' 

*' ^ • . 

Tl^e f^mrulas^ lA t61s paper are developed conditional upon the selection 
^ ^* ' * ' ■ 

fof *PrSmar7"Sa»pli^ig Units C?SU*s); they are also developed specific to a 

particular- domain category, and specific to a particular subject matter area 

within Che package, A symBol defines an entity, while the attached subscript 

4 if ■ I , 

serves to determine Its .applicability/ A block symbol refers to a^ random 

^ ■ • * 

variable; and a script syiaBol refers to a parameter. ^Jln^lly, an upper case 
script s^bol refers, -to the population of all units and the corresponding 
lower case script sySbol refers to an estlipate of the parameter associated ^ ' 

with. p. sample -of these units. ISpeclf lcally,-we define .ii '^..'^ 

-'<■*■-'■ , 
1? ■ proportion of «cerc*ises answered correctly, 

' ' * "X^ 

p-(p) - population ^tsample) proportion of etlg^ile llAEP participants, 



. EC^) population (samplej number of eligible stud* 

The fttrstrpopltlon subscript (a) tci be assoclajted with the above sVmbols Wlli^ 
. . ' " • - Y 

^ refey t*o the total population (o), regular assessment respondents q.), or 

^ ,\ J. . " . — ' 

nonraipondents or Ho-Shows, (2) . Let; Z^^j^ be a domain Indicator variable such 



tliat,.' 



*ajk- 



1, if eligible student-k belonging to response grouj-a 
' In 8c^ol*J Is a meoBer of the specified domain; I 

0, otherwise* * - . * ' 



He then define 



■ / 



0 



and ^ 



TEX'. ■ Z 2 T" - / B . 



,aj j^j^ ajk. aj 



population totals F Cy3c>,'c Cyacl, P Cx), and Cx) will ref ey to th« (juantltlea 

a « q • • 



• 




- I 

jell 




hi Ki' 


a - i[l 






; E 


■ 


hi'°-ai' 


a -1 1,1" 






- i: 




hi ^i'-^ 


a-* 1,2 






- £ 

■ jell 




P n * 
2^ aj'' 


a - 1,2'.; 



- Sample estimates for these quantities will be denoted by'J^(yx), c^(yx), fjjC^)> , 
and c^Cx} respectively. The (jCantities will be combined to assess the xaagnitude 
of .nonresponse bias b^domaia in KAEP regular assessment Statistics* 

The following STmbols are used In the preceding and subsequent r^^j * ^ 
foxpulation: 

h pseudo«*st;cat^» 

i - PSU Within piet^o^stratua, , * ' 

% - -4 ' • ' * * 

' ' k- sttdent within scliool,' ^ ^ ^ • 

. » ' . I 



m. » ntmber of eilglble sample studextts^ taking a package , 

V " package aample nonresponse adjusted vetg&t .Cl*^., iir^erse 
of the proBaBlllty of selecftlon \tnto the study} ^ 

^ ft - set of all eiiglhle sc^^ls, J - ^ ' 

u - £^ple set of ellglBle schools, 

« ^ 

^ ^ ■ sunmatica overfall posslBle subscript values • 

3*2 Domain Estimation Methodology 
- 3.2.1 glrst'Order PSU Estimators 

First, xipte that .the '*true** domain mean Is 



4 ■ ■ 'l«+<!2«- 



The regular assesaoient estimator for YX. is 



S 



/ 



viXh expectation 



Therefore, 



i . 4SS. 



Bias Cyxj^) - B(y*j^) - 'ZX^ 

4^ 



((3.2.1.1) ^ 



Similarly, • <. > . 

i RelVBias (yx^i - Blis (^1 / ^ 



Pj^CyxJ + CjCyx) 



F^Cx) t Ci,Cx) 



'-. 1 



(3.2.1.2) 



Ratio-tTpe e«^lmator» are us^ to eatlaate valties associated with eqtiatloii^ 



3.2. la and 3.2.1.2. If we l6t . ♦ 

* • t 

. r.- f,^I +.p9(yxl» 

* \ X • ^ 



\ 



(3.2J1.4> 
'(3.2.1.5) 
(3.2.1.6) 



thea 



and 



where 



'X 



, . jcM, W 



I 

t 



The parameters P^* and are eatjmatea-ftom. school response rates durtog, 
regularxasaessmcat^ The eijtliate for a Ho-Show rftudy grbup package U found 
as the response rate' to -aU g?oup packages given in that school. The v^^^ 



welglits deapte the reciprocals of No-^Shov selection probabilities adjusted 
for Nq-S6ov nonre$ponse. ' 

* The preceding statistics ^ield domain bias estimates Involving In^School ^ 
• * ' ^ ' * " 

regular assessment respondents and all No«*Shov respondents. Anpther set of 

meaningful domain bias estlm^ttes Involves In*-School regular assessment 
respondents, and m-ScTEiool No^Show respondents. The definition changes liidi*- 
cated by the C*) In equations 3.2.1.7 through 3^2. 1.10 were motivated by an 
attempt to forsi a matched school domain bias estimator iaeed exclusively on 
In-School NorShovsj. The set of schools Q**^^} is the subset of regtialr assess- 
ment schools G*^) which, provided In-School No-Show responses for the particular 
' 2 ; 

package In question; The deleted schools either had no Cooperating^ In-Sc)iool 

No-Show respondents for 'the package > or were subsampled out at the No-Show 

packag^e assignment stage to control the package yield per FSU. The regular 

assessment respondents foaf the set of < schools with In-^cliool No-Show 

• . : ' 1 . . - , ' ' . . 

responses for the package were Inflated to account for the deleted schools » 

^hence the adjusted w^' -weights. Thus, 



u 



- fjcxl + 0*00 , 



C3. 2.1.7) 



C3.2,l,8) 
(3.2,1,9> 
(3.2,1,10) 




^ / ' ^2i *11 * * ^ * 



"2j. 



''2^^^ Jew, ' * . J, ^Ijk ^2jk • ^ 



•m r 



With denoting the mimBer of In^-Sctpol No-SJiow responses fron school jew. , 
the definition of t;^e^t of schools w^^ assures that =*2j ^ • 

3»2»2 O verall Tlrst Harder Estlanates * * ' 

— " * ' 

^ To facllltatja. the cnsulJig discussion, attach subscripts to p, 4, t» and u 

it if '-k it 

of formula 3.2a*3 through 3*2a*6 and p , 4 , t ^ and u of foirmola 3*2a*7 

through 3t2atlO Ctt«*> buBscripts "hi** tb Indicate PStI-1 within pseudo- 

/ 

3tratum-h)» Using these, quantities, onfe obtains t,he overall estljaate involving 
all No-Shovs as ' 



" ^ ZUa^l ' - , .,(3.2.2.1) 

* ' 

and -^^••■'tas(V:x^ ^ ' 

rel*^blAsC7xi)^ft^ , . . C3*2.2*2) 



and/ involving only In-School No-Shows, as 

• ... 

..'*' * * 

^ ' . 'Bla8*&y-%*^-*^ (3.2.2.3) 

and , , . 

*• • ' ' r&l*-^las (jx^ ■ -5 — 5-r - 1 . (3.2,2.4) 



^ i3.2r%3 Second rOrd ex Eatlanattbrs 

The second^order estimators of variance for expressions 3.2 ,2, 1 through 
3.2.2,4 *re, based upon a form of the "jacWcnlfe" techniqiue Introduced by 
/Jyenoullle 17] and advanced for interval estimation by Tulcey 18] . Th.*e procedure 
is presented for domain estimates Involving all No-Sliows, .althoijjgh the procedurte 
for domain estimates involving only In-Scbool Np-Shows is similar. ^ 

First, ve define , ^ ] 
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Since the 57,PSU*s make up a half-sajnple of HAEP regular assessment PSU's, 

the desirable condition, of having twb PSU selections per stratum does not how. 

' , s , * 

Instead, pseudo strata were formed by sequentially pairing the No-Show PSU*s 
according to region andv size, S^ce^the number of PSU*s is odd, one pseudo 



stfatm /Oi.°i was assigned three >Sli'?. The associated jacfckntfe estimators 
of variance are ' ^ , 

E -,23: 2 
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and 
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To assess the significance of the domain bias and. domain rel-blas 

. , ■ r ■ - 

esti^test one might be' willing £o assume that ' 



blasCyXj^) 



_ 11/2 

Jyarjblas 

and . * 

« 

» ■ 
■ * . * rel-bias^x^) 



Jvar|blas 7^ . ^ \ . ^' 



jvarlrelrblas 



for each, dooaltt category are distributed as "Student *s'*,t-3tatl8tlc vlth 29 
degrees of freedom. Under this assumption, algniilcance with a" Type I error 
of 0.05 is indicated when |t| >. 2.045 or |t»| > 2.045* V ' 



4. DESCRXPTtON 0? METHODOLOGY APPLIcAnON " ' ' ; ■ 

-The procedures for cbinputlng the fonmila described in secttou 3 are 

' . * ' ^ 

sxnmnarlz^ In the four paragraphs which follow. A listing of the software 

developed to execute these computations is included as appendJ^B. The ^ 

calculations described in the fallowing paragraphs are performed for each 

4Io-Show,group^ pafckage Clumbers 1, 3, and 9) and for each subject matter area 

^ * / ' 

(mathematics and aci^nce) within, ^ch group packager. Befoije any production 

runs were made, aJ^ calculations were verified by hand. ^ 

The computation ipfocess followed several s^eps. First, those components^ 

of p, a, T, 0/ p*, a*, T*, 'snd o* deactihed in section 3*2.1, which pertain to 

regxilar assessment respondents , and No-Show r^^oto^^i^s, were computed separately 

. for .each PSU and for' each domain category. Second, the quantitlJes described in 

> * ' it it ^ it * 

the preceding paragraph were combined to foi^ P^> t, p; p , a ,'t , an4 o 

for. each PSU and for each 4omaln category. These quantities vt^re summed (^er 

allJfSU's fo"r eich domain, to generate p_^^ a_^> x^, o^, p^, t^, and 

0 described in ^tion 3»2»2. Thir4>^ bias and rel-bias estimates wfere cal-i 
++ . > . • • , ^ 

culatcd for each doaaQ category for all Ho-Shows WLd fox I;i-School No-Shows 

only.. These --(luantltiea have been previously. described Infection 3.2.2., In 

additlbn, 6^,^ 0.^, y^,, , and of section 3.2^ wete computed for ea<ih pseudo- 
nl , 02 Ai ■ nz I 

stratuBi-hi, for each domain category^ and for all No-Showa and for In-School 
No-Show^ only. - Finally, the estimated variance of the bias and the rel-bias as 



described in section 5.2.3, was obtained for^all No-Shows and for ^School 
No-^Shows only for each domain category. \. > ^ 



* ' 5 V* NO*i3H0W DGMm ANALYSIS. cdiJcLTJSlpNS 

i. /I ' 

5.1 Tablias StimmarlglAfg Results . * 

The results of Ho<^Sbov domaJja analyses are summarized in appendices 
C an& Df Average per exerdise estisiates of bias and relrbias by.domaidcsu^in 
category are prorvided by paclcage and subject natter area for all No-Shows 
^fixid In-^School Ko^Shovs only in ta|)}le's C«l through C«6r* The variance of 
these bias and rel-bias measures is also provided in theu tables* Using the^ 
t-test described in section 3t2,3r those val^eT^ the bias and rel**bias 
which are significant with probability of a Type I efrot of\0-05 were 
denoted by an asterisk.^ / ^ . ^ 

Weighted estimates of the mean numbel^correct responses per exercise 
by doaailJ^re provided in tables C.7 through C*12. These ebtimates are 
provided for ^regular^ assessment Respondents as well as for^*Schodl and 
Out-of«:School Ho-Show respondents'. Results are further delineated by paclcage 
and subject matter area« *The fornojla used to compufe these estimates was 



jm„ b.1 ""^^ 



a 



rates 



Tables tit£3 throu^Ct^S provid^e weighted estimates of response rat 
by doOain fpr regular assessment respondents, these estisiates are provided < 
^parately by paclcage^ The formula used to compute thebe estimates (ms 



• ' ^ - " * ' • 

X : ^ ij * ■ 

' < ^ jctt^ W ^J'^ ^^"^ ^ . ^ 

1 

Tables D^l and D,2 provide estimates o^the laean number correct responses 
by package and subject matter area for Age Class 3 Dropouts and Out-o£-School . 
No-Show respondents, respectively, 

Table D*3 .stimmarlzes by domain tbe number of times the rel-blatf for all ^ 
No^Shovs was significant and the number of times It vas negative by subject 
matter area» The rel-blas measures are also ranked by size over all three 
No-Show group packages* 

Lastjiy, ^table D*4 presents the*«verage Out-of -School percentage contri- 
bution to the total >iast Tlia a:verage Out-of -School percentage contribution 
to the total bias was obtained as follows* Plrst, the Out-of-School component 
of the bias w^s computed for each of the tfccee No-Show group packages by 
subject natter area/ as ^ * . ' 

, " ,__Jji-Scho_ol O\it-of -School^ 

component ^ component 

where ' * 

w " tlie average^^roportlon of la-School No-Shows; , 

Y " th^ average nttmber of' correct responses for regular assessment 
^ respondents^ * - ' 



V 



« 



ERIC ■ 



yI-. » the average nxnaber of correct ..responses for In-School lIo**Shows; 
^20 " average number of correct responses for Out-of-School lIo-Shows# 
Second,, the Out*^f-School percentage contribution was computed for each package 

1 ' . r • 



as 

Ouf*of-School ^ dut'^of-School coaponent j^qqjj 

percentage la-School component + Out-of -School component 

Finally^ the average Out-of ^School percentage contrlbubitm was obtained hj 
averaging the preceding ({uantlty over all three group packages In the Ko-Sbov 
study, 

m mm 

5,2 Discussion of Itesults ^ ' ' ^ 

* 

As expected y the number of significant biases ^ong domain estimates of 
the various bias measures In tables dl through C«6y Is greater when all Ko*- 
Shows are used than when In-sSchool No^Sbowa only are used. The preceding 

statement la true for both bias and rel-blas measures. There are sllghtjLy 

^ / 

more significant valuer with, mathematics than s^clence exercises within packages* 
Package 1 exercises tend to yield slightly more significant values. ^ Since 
the performance determining exercises are assigned to packages arbitrarily^ ^ 
it is not surprising that the level of hias would vary from package to package. 

Few consistent significance patterns eme rge ^wi thin package and among 
domains. With respect to rel*-bias estimates, when all Ko-Shows are used, only 
Rac&*-tfhite exhibits significance with all combinations of subject ma^tter and 
packages* Conversely > Race—Other and DPE — Some*High*Schoc^ exhibit no slgni- 
fic^ceSf Results are no more consistent with corresponding bias measures. 
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The^Mmitude of the per"exftrcls& bias measure Indicate that, In 'general, 
among dqmains, bias for mathjematlca exercises Is slightly greater^ than for 
sclenf:e exercises. Xaconslstencles of this pattern are often attributable to 
smalS^ sample 

.^'Several patterns In magnitude emerge within domain^jmrlable categories. 
These patterns tend to be more pronounced with science exercises* ?or example, 
wit;jrrel-blas and all No-Shows in science. Sex— Male biases texid to be greater i 
than Sex— Female biases. Region— West biases tend to be* largest and Region— 
Northeast tends to be smallest. Race— Black Is ^rge^ while Race— White and 
jjace — Other are smaller, STOC— Suburban-Prlnge tends to be largest and STOC— 
Innter-CltT-Prlnge smallest for science exercises, DPE— No-Higt-School tends 
to be the greatest while DPE— Post-BigL-School tends to be smallest, 

fhe ^OC patt«ns for mathematics exercises are more intuitive with Inner 
City. Fringe apd Small City yielding^ the largest biases and Subxirban Fringe the 
smallest, iow Hetra has relatively large biases for both science and mathematics 
G^ercisfcs, Except J^or STOC, where the No-Show sample sizes for the extreme 
iategorles are very small, the above-mentioned patterns remain essentially 
intact for co.rxespondlng mathematics domains, A suamary of some results from 
tables CI through C,6 is found in tabJte D,3 

Negative estimates tend td appear periodically bat without much conaia* ^ 
tency ^th respect to domains. Negative estimates of bias measures, using only 
/in^School No-Shows, >tend to appear ^most frequently in STOtJ categories and ai;e 
generally nonsignificant and small in size. Since sample sizes are often rel- 
atlvely 9mall under these circumstances, one mlghf attribute the negativeness 
to^ sampling error, - ' " . - \ 

* 

• 30 



When one observes the; results In tablw C.7 through C.15 juxtaposed 

* > 

vlth those?ln tables C.l through C*6, It becomes readily evident that the 

magnltfide of doQaln biases In the later tables Is directly related to the 

domain-specific difference In regular assessment and Ko-Shov performances 

and ^Indirectly .related to domain-specific response rates. ^ Once agaln^ ..r \ 

small sample, alzes may dlsruptr the. consistency of such^ a statement. 

By ob^serving tables D.l and D.2 one notes that the perforotances of 

Age Class 3 Dropouts and Out '^of -School Ko-Sbows are virtually the same^ for 

^11 combinations of exercise subject matter and packages. This has admin- 

Istratlve Implications which ^a discussed subsequently. _ « * 

' . ' , \ ^ ' 

O^t-oS-School Ho-Shows present a major *contri&ot;ion to the total bias*. 

Results of table D.4 shov that, in Virtually all domains, pat-bf-School 
ft 

No-Show8 make up a small propoiftlon (1-W) of all No-Shows, and yet they: 

a^e responsible for, a large share of the tt>tal bjbts. Since the results of 

tables D.l and D'.2 Indicate that <Xit-o£-School ifo-Shows and suppleaentiry 

frame' DrtStouts perfoia "similarly on the three group packages i HAEP night 

veil adopt, a n9nresponse imputatdLon procedure in which Dropouts assume the 

performance of Out-of-School No-Shows, or vice versa. AdmlAlstratlvely, 

this implies that either Efcropouts or Out^of-School Ho-Shows yould be followed 
*. « • 

up and test^ in order to reduce nonresponse bias. Assessment performance by 

imputation could be done in any of several ways. Subsequent analyses will be 

designed to investigate the effects of these imputation procedures on the 

magnitude of nonresponse bias* These analyses are discuss^ in section Tbi 

results dlsctlssed above would tend to stipport othez^ findings in which the'fnag** 

• . ' ^ - : • ' 

nltude and number of significant results associated with biases are reduced 



when, only In-School No-Sbows are. U$ed In the estimate. ^ 

The size of ^ the percentage of the Out-of-School contribution (see 
table D-4) In those donalns with significant bias, does not seen to conforn 
to any particular pattern when the bias is measuiie^ and all No-Shovs are used. 
However, when rel-blas is considered and only In-School Np-Stiows are used, 
the percentages associated with significant domains (end to be^rel&tlvely 
smaller, particularly in science domains. These phenomena would Conform to ^ 
intuition since significance with only In-School No-Shovs would require a 
relatively large contrlbut^^ by In-School No-Shows jgverall (i.e., low (Xit- 
ef-School percentages}* 

Differences among average Out-of-School percen^es within domain 

. variables appear to be more heavily attributable to d£ff ercnces in W (pro- 
portion In-School Ho-Shows) than to absolute performance differences. This 
is observed In table i>#4 by noticing a high negative correlajclon between the 
estimated averj^ package value of W and the'average Out-t)f-School percentage 
The negative yelatlonshlp czn be explained froa the formulu used to calculate 
the Out-of-School, ferceatage, Further details pertaining to this fornula have 
been presented In section 5.1,' For fixed Qt^ - Yj^) and - Y20),^a8 W 

, Increases the Out^f-School contribution percentage decreases, ^or exasple» 
^ere the average Ofit-of-School percentage for STOC— Low-Hetro Is high, the 
corresponding value of W Is relatively low* 

The -average of Out-of-School bias percentage^ differs markedly only with 

* " * 

corresponding STOC categories betweCfr mathematics and science* This may be 

due to small sai^le sizes In STOC categories (see table A.A)* With other ^ ^ 



i 



■'■ > , • / • • • 

doaain* variables difference* between correspcmJing mathematics and science 

«»■ • » ^ ' 

domains are ^5sall# 

• -fc 

5*3; "Recommenda t ions , • - ^ 

Two basic reconmendat^oni ealnate from the findings ot this study to 

T ' > ' ' " ^ I 

date.' First, since a Oarge portion of No-SJiow bias is* attributable to the I, 
Out-of-School Ko^Showa and since the'V«f ""laJic'e of these Ho-Shotfs appears 
, to be siallar to D^ofout fraae respondents, HAEP may wish to investigate the 
possible use of Dropouts to compensate for bias of estimates t^hat is attril- 
butable to Dropouts or Out-of-School No-Shows. Xhi^/nvestlgation' could be 
in c6nnection with an Investigation of Imputation procedures which are dis- 

cussed in section ^'KN - ^ ' 

Second, 'to rcducfe the number o*f In-School No-Shows, it is "recommended 
that several alternmve In-School followup procedures be Investigated and 

compared. This recom men dation arises from the fact that supplementary data 

• , / " - ■ 

have Indicated that a lirge proportion of _ No-Shows had legitimate achedulinS' 

difficulties because of other sShool-r elated activities or tlljsesa. These 

ffi^lngs ire reiniorced by the "Reason for No-Show" verification performed 

• by*tihe J)istrict Supervisor a^er nonresponae. Xhoie 

In table 5.1. Perhaps greater^ ,f lcxlblll,ty in scheduling regular assessmenij , 

■ . ' • ' / 

administrators would realize a greater response among In-Schdol No-Shows. 

To iapl^ent this recommend atdoU/ a ^al\ pilot study will be conducted In 

connection with Year 06 dat» collection to test results from a number of alter, 

native followup procedures. This study is mot* fully" described in section 6.1, 

, • • •*,. 

It is strongly recommended that every effort.be made to maximize th^ 

proportion of initially selected students that receive their assigned regular 
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Tabic S.l "'"R^SonJjfor^Hd-^Show" codes 
Cvertflcd by D.S'.^^hortly after Jiont^epsonse) 



Reason 



1 

ft 




2 * , 


3 , 


4 


• 




J 










' (Z.6Z) 


31 


• (3.6X) . 






9 


(I.IZ) 


22 


14 


(1.0*) 


23 

• 


(2.9?)' 


Id 




17 * 

• 


/. • 


10 


' (1.2Z) 

« 


4 

• 

9 


(1.2Z) 


12 




25 


(3.0Z) 




(2.9Z)' 


5 


(0.6Z) 


6 




J. 


(0.9Z) 


. 0 




5 


-(0.6Z) 


7* 


(0.8Z) • 


4 


(0.5Z) 


10 


(1*1Z). 


2 




0 


TO. 7*) 


J 




19 


(2.2Z) 


3^ 


(4.2Z), 


27 


(3.2Z) 


9 


(I.IZ) 


10 


(I.IX)- 


5 


' (0.6Z)' 


. 9 


(i:iz) 


13 

• 


'(1.6Z) 


3 


(0.4Z) 


0 




9 


. (r.iz) 


26 


(3.1Z) 


178 


(20. 5Z) 






64 


X7.6Z) 


173 


(20. 9Z) 


173 


(19.9Zi^ 


.290 


. (36. 9Z) 


201 


(23. 7Z) 


250 


(30. 2Z) 


346 


(39. 8Z) 


ZJ7 




365 


(43. IZ) 


173 

• 


(20. 9Z) 

• 


49 


/ (5.7Z)v 


90 


(11.5Z) 


Ul 


(13.1Z) 


id2 


(12. 3Z)* 


0 




2 


(0.3Z) 


^ 4 


(0.5Z) 


3 


(0.4i) 


8^8 (100. OZ) 


785 


(lOO.pZ) 


846 (100. OZ) 

• 


827 (100. OX) 



-Total 



I^ate for session 

Extra currlctilar 
activity ^ 

Exam or Important 
^ class 

Notified but forgot 
Uent home sick 
I^ate for school 
Work' study program 
Report for work^ 
Cannot remember / 

Other • 

■» ^ 

Absent^ from school 
entire day ^ 

Unknown 

withdrawal from 
school 

BlanU ' 

TOTAL 



82 (2.*5Z)» 

70 (2.1%) 

56 * (1.7Z) 
43 (1..3Z) 
16 (0.5Z) 
21 (0.6Z) 
88\y[2.7Z) 
37^(1*1Z) 



38 
479 

914 
1,121 



' (l^Z) 
(14. 4Z) 

(27. 5Z) ^ 

(33 . 73ftr I 



352 (10. 5Z) 
9 (0;3Z) 



3,326 (100. 0;() 

'—~ 



/ TabUS.l (Conclnued) 
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* 




* • 
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• 






t 








Region 


• 




Reason 




1 


2 


a 




t XyboX 


/ 

Vacation *q 




0 


• 

0 


0 


•I 

1 


1 


Ncj, Show 




20 


0 


0 


' 0 


20 ^ 


DlflchairlCEed 


« 


7 


> 0 


0 


0 


7~ 


Non-*En<rl Ish 


J 




0 


0 


0 


6 




; 3 


0 


* - b 


0 


3 


Refusal 




69 


5 


11 


10 


95 


Transferred 




' 4 


5 


fl ' 


; 1 


18 


• 


t 


11 


15 


^ 13 


' - 18 . 


57 


Assessed 




• 2 


2 


0 


' '3 


• 7 


Hoiaebound 


• 


1 

• 




4 


3 


12 


t 


* 


1 


3' 


2 


■ "11 


17 


OUDMC&fUiSU 


• 


0 


6 


• 0. . / 


1 


7 


npddtifttfid 






. 4 


2 

<• 


» 2 


8 • 


T Tn If n oun / It 1 A nlf^ 

uniuiuiwii/ oxQiifc 






18 ' 


24. _ 


*1I9 • 


. 215 -. 


Hoc enrolled 




ft- 


1 

• 


0 ' 




1 


Trade school 


» 


0 


1 ■ 


0 


0' 


. 1 


• 

Gheck**ouc 




0 - 


0 


0 




A 
H 






178 


64 


64 


hi3 


y9 



assessment admlnls.tratlons . This recommendation Implies that special efforts 

t * i A 

should ^e made by DS*d to Insure that school personnel notify each selected' ^ 
student of the date» tlme» and place of the administration «r.^ Exarcise 
Administrators should allow for greater flexibility In scheduling admits- 
trations tp allow for possible^ xfoxifli^ts with other school-delated activities. 
^*This procedure would ^lud.e waiting a few extra minutes for iatc initially 
selec;jted respondents to show^up, 



'\ . -6. EXTENSIONS 

7 * 



SevcrS otfifiT aimly?es could be \2n<i^rtakexi*'ln connection with the 
overall goals o£/the Ho-Shov^tudy. The purpose of this section Is to 
preview some of these analyses* . ^ ' * ^ 

6*1 Separate Exercise Analyses 

A;ial78es of bias up until this point bave been formulated at the 
level of a Student ^ s^ o verall performance on the group olT mathematics or ' \ 
science exercises*" ^Rd in each of the No-^Shov packages* As presently * 
conceived, the large nymBers of exercises ^rfilch malx up the packages wotild 
require that the scope of these analyses be limited to the total populations , 
and that if domain analyses were done, they* would Involve only a few *1cey^ ^ 

exercises « - - . ^ 

• - / ♦ 

Methodologically, the procedures and algorithms, yfaich are iised' for 

three separate exercise analyses, will be similar to those used for previous 
analyses if the blab of the'mean is considered # The difference is that the 
measurement variable is dicliotomous, under these circumstances,' since, for a 
single exercise, the proportion of correct .responses (7* to follow the term^ 
Inology of the present and past Ho^Show working papers) would be either zero 
or one# By this approach the bias of the mean proportion of correct responses 
woi^d correspond to the bias of the proportion o^ correct responses Co the 
exercise among all eligible students. 

Analyses at this level coxxld conceivably Indicate differences In che 
nagniCude o£ biases, among exercises. This Information may lead to plnpoliitlng 
exercises that contribute most heavily to the bias In package by subject matter 
scores f If some separate exercise bias analyses were done by domain, the 



prloary source or sources of bias might be further pinpointed. 
6t2 Jestqi' Effects and Power . / 

Two supplementary evaluative measures of analyses are the sampling design / 
effect and the power Cor sensitivity) of the tests that are performed* The 
former indicates the precision of estljnates from the study's sampling design ^ 
relative to a design with, the same sample^ but where simple random sampling Is 
performed. The jatlo of the variance from the present design to the simple 
random sampling design Is known as the design effect. T&e latter measure Is ^ 
defined as the probability that a significant bias 1^ detected by a test, given 
tbdt the results are of some arbitrary degree of significance^ 

Since the comMtatlon of all analyses provide for computing approximate 
variances of all estJSiates according to the present design^ design effects are 
computed by obtaining approximate variances o^ the biases and rel-blases according 
to the assumption of simple random sampling* The estimates of biases and rel- 

* 1 * 

biases are represented as coioblnatlons of various products , sumSt and ratios of 
random variables which implies thati even under the simplifying sample design 
assumptions 9 the variances must be approximated. As one possible set of approxi- 
mations for the above, if x and y are estimators and E(x) « X and E(y) ■ 7, then 

VarCx + 7) - VarCae) + VarCy) + 2Cov(x,7) 

VarCxy) - X^Vat(y) + ?Wcx) .+ mCovCx.y) 

* Var(y/x) - \^Var(y> +^VarC5c) ^ • > . 

Since and Y wlll^not be knovni x and y, re^^^tlvely, can be used in the 
above formulations • . . . ; ' 



The power of hypothesis tests of bias,^ subject to certain slmpliiylng 
distributional assumptions and levela of significance, can be Illustrated by 
compiling several tabl^ where precision measurers vary among the tables and the 
magnitude of the hlas^ ^^se sl^olflcance is to be detected by the ^est^ is 
varied within the tables , The power or sensitivity Is ascertained by ^ 
observing the estimated bias magnitude (assuming it was the true bias being 
tested as equal to zero) and the .variance of the estimate* 
6*3 Itoputation Procedures 

I- f m 

Imputation procedures present a means by which Mases aptr\butable' to 

nonresponse can be reduced sometdiatf but never eliminated completely* If pripr 

- ^ ' ■ i * 

findings Indicate a small overall bias, imputation procedures may siifficiently 

reduce the bias^to levels where major analytic studies of nonresponse bias 

would no longer be necessary* . Assuming thl^ scenario to be a realistic one^. / 



V jBJx Investigation of various useful imputation procedures may be rewarding* 

Several approaches are p»aible« If it were found that regular assessment 
nonrespondents are comparable» by ^ome definition, JLo regular assessment respondent 
from the dropout-or early ^aduate supplementary frames, thea any nonresponse^^ 
bias could be reduced' by weighting the supplemental frame respondents more 
. heavily to co]iq)en8ate for the data lost to nonresponse* For this reason, a 

coisparlson of supplementary frame and Ho-Sbow package data has been done* Several 
comparison combinations o£ supplem^tal frames with In-School or Out*of School 

V Ho-*Shows ^re possible and have been fabul^ted (see^appendix D). The outcome 
♦ - 

of these and pthftr similar «^[ialyses may yield a model by which accurate 
estimates can be made from regular 'assessment response data flone, thus 
obviating the need for^nonresponse folj^gwup* Another weight adjustment 



technique Is the so-called hot-deck procedure which yes used for th8ri960 

Census. In this me/hod, cross-cjAsslflcatlon cells are formed hj one or 

more- relevant variables available froa both respondents and nonrespondentfl . 

lor example, soiae of the auxiliary data variables could be used for this purpose. 

Each cell is suppldLed with, a single response. from among the respondents. As the* 

file of data is read, new responses for the cells are supplied as they are 

encountered. These responses remain until another response from' the same cell 

appears. Whan a nonxes5>on8e Is encountered it Is identliied with a cell. The 

weight of the' nonrespondent Is then attached to the tiurrent respondent occupying 

the cell. ^Ihen several relevant variables are availa^ift^r both respondents and 

nonxespondents^, the Automatic Interaction Detector subroutln^^k^n be used to 

specify which cross-classifications of these variables should be used to define 

hot-declt ceils or weighting classes for the substitution method- mentioned below 19] 

Finally, a weight adjustment procedure Involving the use of multiple regression 

may be used. With this- technique, the binary response variable is regressed on 

some set of relevant variables which are available for respondents and non- 

•-»- • -> . 

respondents. The nonresponse adjustment fot each respondent is the Inverse pf 

the fitted response variable, ^e usefulness of the regression approach is con- 

tingent'upon a high degree of association between the auxiliary and ^ response rates. 

This approach is similar In nature to the so-called Politz scheme In which the 

'adjustment is the reciprocal of a respondent's probability of being found. ■ ^ 

Imputktion by substitution of additional selections frpm the populStipd 

(i?^., aftemates)'iB currently used for Individual nonresponse In HAS. A 

selection of additional, students Is made within schools. Th^se students are then' 



selected to replace students originally selected buj: who fail to show up^ In * 



; addition to jTailing to^ yield unbiased*estim£ttes, it should lie noted that 

rj;>adjustment techniques tend to roduce the precision i>f_ estimates, assuming alT ' 
^else equal,. ^ * • * ' ' ~ 

^ Several ^of the^e Imputation techniques coulcf be tested and coiiq»ared» The' 
criterion of coiiq)arison would be th6 mean squared ewror of these estimat^.^ 
Thi'g criterion- i*s coirauted ad the, variance of ,the estimate pltis its bias 
^ squared. The variance can te found directly and tW bias, can be estimated xisingj| 
"true" paraaieter values estimable from re^iap assesament and No^Show sai^le 
data. Data- manipulation techniques like the imputation procedufe* mentioned 
above ar^ altematdves to more cpstly followup surveys aimed at No-Shows^ . A ^ 
' cost model can be developed for a followup sample of No-Shows to Judge tl\e - ^ 

% : . ... 

relative cost and mean squared error efficiency of survey versus, data manlpu- 

s ' 4 
* Xati6n methods • * ' . * 

6^ Pilot Study . ] , * r . • 

^ 6.4.1 Eacperlmental Design ' ' . / 

* • * The purpose of this pilot study wltl be to investigate two novel 

\ - ^ . ^ ^ . / • ^ ^ r-. 

approaches which could be used to Increase the proportito of Initially sel- \ 
' *. ' . ' . > .J* 

ect^ students who are a^inistered packages.^ Schools will be categorized 

« i 

X as one;, of two types, according to the number of days required for assessments ^ 

One category will involve thbae schools requiring one dhy ot less an^Jrthe 

«^ ' . ^ . ^ ^ • ^ ' ' 

• other* will involve those rcfqulrlng more th^in one^dgy»^ One of the approaches 
that; Vill be^^tested (l.e,, design trpatmentp) dictates that No-Show followup 
i& done and* that makeup session (s) he arranged on the same day as the scheduled 



adalnlstratlon. The other approach dictates that followup be done and that 
makeup sessions be arranged on a subsequent day. For completen^, these 
two app^^ches will be competed with the present approach used by NAEP 
^4^ . J thfe -control) . Group sessions with sixteen selections per session will 
be u5ed to reflect Year 07 sampling procedures, 

'6.4.2 - Statistical Quality ' 

S 

to ^sess the level of precision that one should expect from the No-Show 
followup pilot study proposed in section 6.4.3r, the deaigU layout in table 6.1 
has been considered. // 

Table 6.1. Layout of No-Shoy followup pilot study 



.Blocking factors 



Package 
assignment 
load 



Expected 
re^onse 
level 



Control 



Sample 
schools 



Response 
rate - % 



Followup procedures 



Same day 



Sample 
schools 



Response 
'rate - % 



*Next day 



Sample 
-•scbodls- 



Response 
rate - X 



• One 
day 
assignment 



Low*. 
High 



6 
6 



68 

78 • 



3 
3 



78- 
84 



3 
3 



85 
88 



' Horft than 
f one day 
assig^ent 



T 

Low « 
High 



6 
■6 



68 
78 



3 
3 



7V 
4184 



3 
3 



■ 85 
. 88 



Notice that a blocking f actdt which categorizes ichools according t9 ^helr ^ » ' 
anticipate level of . response l^as been Incorporated into the layout above. 
Anticipated low response rate schools wUl consist roughly of- Low Metropolitan, 
Urban Frlngtf, and Suburban Fringe schools. High response rate, schools will be ' 
drawaJ^am the High Metropolitan, 'Hedlun Clfy, SmiU Clty^ and Extrene- Rural 

STOC categories. . , , . i , , 

/ Implicit in ta^e 6.1 is the assumption that there wilt be little or no 



difference in resBonse rates between schools with k one day package assign- 7. 
nent and those with a two or more day dsslgnmen^. The rfesponse rate figures 
assumed for the control group.s are typllal of those observed for Year <W ^ 
-17-year-olds In tablfes C.13 through C.15, of appendix C* The response rates 
set -for the same day and next day followup procedures reflect the level of 
Improvement that we would hope to achieve* The res^nse rate layout in 
table 6.1 also assumes that the next day procedure will be superior to the 
same day followup and that the improvement .among high response rate schools 

m 

win not be as great as that among low response sic^ols. 

The statistical power of the pilot study design proposed above will 

depend on the precision , of the estimateti response rates. Suppose that 1 - 1, '2 

denotes the low and high responae'rate schoolsj.j - 1, 2, 3 denote respectively 

the control, same day,, and next day followup procedures; h - 1, 2 depicts the 

one day assignment and cwo-or-more-day assignment sch^Xs; and k - 1, 2,..., 

8(ijh) indexes the response level-1 schools assigned treatment-j. Suppose 

further that r(ljhk) denotes the response rate among the n(ihk) students 

selected for assessment in school-ihk and*'followed up with, procedure-j . The 

response rate for treatment-j among response level-1 schools will be estimated 

> ' 
by • ■ . 

r(lj) ' t ■ Z* r(ljhk)/2s(ljh) (6.4.2. U 

h-1 k-1 

* 

If R(ijhk) is the response rate. that would be obtained from all the N(lhk) 

... ^ . 

'eligible students in schoolrlhk when treatmpnt-j is used, then the expected 

-^alMe. of r(^jyover successive student selections and randomization of tflreat- - 
ments ^.s ' . ' ^ * ^ 



'2 s(ih) 2 
( . R(lj) - £ 2 -R(13hk)/28(lh).- Z R(ljh)/2 (6.A.2.2)- 

: , h< k-1 " ^ h-1 

Where g(lh) ■ 12 for all ih pairs since there are twelve schools in each of 

our four experimental blocks • The response rate figures in table 6.1 repre-. 

spnt^ hypothesized values of the R(ijh) parameters. Ignoring fliiite population 

corrections and assuming t;hat the student sample sizes n(iHk) - nCh) for all 

• 24 schools in package load block-h, one .arrives at the varianca expression • 

' ^2 n-^(h)/2[ ^ 



. 4(ij) 

where' ' 



' 2 ' s(lh) ' • 0 

'<P: ^ (il)" / - • 2 - 2 [R(ijhk) - R(ijh)r/Z;*a^) - 1] ' 

-^^>-C-iif\ - 2' .2 R(ijhk) tl - R(ijhk)]/2s(ih) 



"and . • \ ^ 



" /anG ■ ► * ' ' A ^ 

A kij) - ^(iJl) + 8(«2) - 12 for J - 1 and 6 for j - 2 or 3. 

■To prpdnce va^es of these components consistent with the R(lj) values hypdth- 
esir^ in table ^.1 w'e have used the model 

6^ (Ij) - .99 xR(id) [1 - . . I 

std ■ , « ' 

and^ / , ' - * ' - 

•Fdr'one day assignment schools T^e assume that the average number of packages 
assigned Is 2 leading to n(l) - 2 x 16 - 32. For two-or-more-day assignments 

• the average numbef of pacluiges was sVt to 6 resulting in n(2) - 6 x 16 - 96. 

' ' ' / ■ ■ ' 

- The harmonic mean^d'f these student sample sizes is 



•39- 



These consider a tlon9 lead ns to /« 



ERIC 



7ar(r(lj)} 

- (,030625) RCli) [1.- R(lj)l/Saj) ' * (6.4,2.4) 
TabU 6.2 displays il(lj) and associated V(ij) - Var{r(ij)) values along 
with contract coefficicgts C(ii) for 5 islngle degree of freed<?m orthogonal 
contrasts amongl the rClj) • " 

* Table 6,2, Response rates, variances; and contra^td 

• ■ , * if 



Kesponse level 


^. Low response 


schools 


HiRh response 


schools . 


Followup method y 


Control 1 Same Day j Next Day . 


Control j Sane J)ayj Hext Day i^' 


Hypothesized RCy ) ' 


68 


78 


85 


- 78 


84 

■ 


88" 


School sample s<i^) 


12 


6 ' 


6 


12; 


6- 


6- 


Variances VCij) 


5.55 


8".7j5 


'6 -.51 


4. 38 


6.86 


* 5.39, 


Same vs.* Next' ^ 


0 


-1 


+1 


0 


-1 


+1 


Followup Control' 


-2 


+1 


+1 


^2 


+1 


+1- 


Low vs«. Rlgh 


-X 


-1 ■ 


-1 


' +1 


+1 




(S vs. N) X vs. H) 


0 


-1 


+1 


0 


+1 


-1 


(F-0 vs. C) 7^ (L vs. H) 


-2 


' +1 


+1 


+2 

• 


'■-1 


-1 



To calculate the po^er of studcnts-t tests for the five orthogonal contrasts 
outlined above, We need the noncentrality parameters 



^•23 

4 - I ? t c(ij) s(ij)! 
i-i j-i 



and the degrees, of freedoa (df) available for estloatlng the denooiuator oj^ 
(6.4.2,5)* . . . " . - • 



,45 



tettlng 



I(iJ) 



1. when C(ij) i« 0 
0 otherwise 



we- will use 

V 



2 ' 3 

I I -iCtj)[sCij) - 1] • (6.4.2:6) 



Thcsft considerations lead the power calculations (two-sided » size a « -0 

« 

suimarlzed in table 6.3. , - \ 

■ 

Table 6 •3. Power of two-sided t-tests 



Source 


|CRl 


2^CR 


1 


V 


a 


6 


Sane vs. Next 

* 


11 


7.48 : 


- 1.48 


20- • 


.05 


.50 


Followup vs. Control 


43 


11.60 


3.71 


42 


.01 


.99 


Low vs. High 


19 


8".65 


2.19 


42 


.05 


.82 


(S vs. N) X (t V. H) 


3 


7.42 


0«41 


20' 


.05 


<.10' 


Of-U vs. "C)- X (t-vs.. H) 


11 


11.60 


0.95 ■ 


42 


.05 


*.30' 


M ^ 















Table 6.3 dcaonstrates that the .power for our primary followup versus control 
contrast should be very good. The Low response versus Hish response centrist 

has adequate power, the Saae verlus.Ncxt contrast is weak, and the interaction 

s 

contrasts have very little power. , • , » ' 



